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EQUIVALENT VALIDITY OF A COMPLETELY COMPUTERIZED MMPI 

Robert E^ Lushene, Harold 0*N^11, Jr.^ and Thomas Dunn 

Florida State University 

ABSTRACT 

Within the context of a counterbalanced design, 63 female students were . 
tested with a computerized MMPI and a group booklet mode of administration* 
State anxiety was measured before and after each testing session. The computer- 
based MMPI scale scores were shown to correlate as high or higher with the 
booklet administration scores than correlations reported for comparisons bet- 
ween booklet and card form administrations or booklet-*booklet administrations 
for a college population. When compared to the booklet version, the computer 
mode initially produced relatively high state anxiety levels. By the end 
of the test, however, no difference in state anxiety levels between the two 
modes of administration was found. ^ 

I 



IQDIVALENT VALIDITY OF A COMPLETELY COMPUTERIZED MMPI 
Robert E. Lushene, Harold F. 0*Nell, Jr., and Thoma? Dunn 
Florida State University 

There have been relatively few investigations of the reli&bility and validity 

/ ' 

either of intelligence or personality tests in which automated equipment has been 
used to actually administer, score^or interpret these tests (Hansen, Hedl, & 0*Neil, 
1971). In the area of automated administration of intelligence tests, Elvood (1969) 
investigated the effects of an automated testing booth for presenting the Wechsler 
Adult Intelligence Scale (WAIS) (Wechsler, 1955). Although the average interval 
between the two testing sessions was 110 days, he found a correlation of .92- be^ 
tweeh the prorated performance intelligence quotient (IQ)' scores based upon three 
subtests. In another study (Elwood & Griffith, in press), thirty subjects were 
tested with a mean of 9.4 days between the two administrations of the automated 
WAIS. The test-*retest reliability coefficients were shown to be .97,. ;95, and 
.98 for WAlS verbal IQ*s, performance IQ*Sy and full scale IQ*s, respectively. 

If one considers the automated version of a test to be an entirely new- 
test rather then merely a type of administration, then the correlation between 
the automated test ^nd the tfa'dit^ional test would be considered an equivalent 
validity index. Several studies of this type have been reported for automated 
tests. Orr (1969) compared the automated WAIS with the traditional administration 
of the WAIS. In this study, using a mean intertest interval of 9.2 days, a test- 
retest coefficient of .93 was reported between the full scale IQ as administered 
be examiner and as determined by the automated administration. Hedl, O^Neil^ and 
Hansen (1971) have developed a computer-abased administration and scoring system 
for the Slosson Intelligence Test (Slosson, 1963). Within .the context of a Latin 
square design, they report: a correlation of .75 between the computerized and 
the traditionally administered Slosson Intelligence Test. The lower equivalent 
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validity correlation in the latter study is probably due tc the fact that for the 
computerized Slosson Intelligence Test the computer also scored the test, whereas 
for the WAIS several verbal subtests were hand scored. Computer scoring of natural 
language input is often less reliable than human scoring of such input. 

With regard to personality tests, computers have been widely used in the 
scoring and interpretation of personality inventories » particularly the Minnesota 
Multiphasic Personality Inventory (MMPI) (e.g.. Fowler » 1969). However, there 
has been relatively little use of computers to administer such tests. One example 
of such an approach Is reported by Dunn, Lushene, and 0*Neil (in press) in which 
they demonstrated the feasibility of computerizing the administration, scoring, 
and interpretation of the entire MMPI. The validity of such an approach is reported 
in this paper. 

It is particulary iitportant that equivalence validity studies of automated 
tests be conducted. Although the very nature of the automated presentation in- 
sures a more standard stimulus presentation than do other methods, certain evi- 
dence su^j^ests that test results may be somewhat different under automated conditions. 
Affective and subjective attitude reactions to computer-based administration of 
tests may differ from those obtained during standard administrations. For example, 
a number of investigators have reported that subjects tend to be more open and honest 
in responding to a computer, particularly if the stimulus material is of highly 
personal nature (Smith, 1963; Evans & Miller, 1969). 

However, in the study by Hedl et al. (1971), in which a computer-based 
administration of an intelligence test was compared to a traditional administration, 
It was found that computer administration of the test led to a higher level of state 
anxiety than did the examiner-administered test. State anxiety (A-State) has been 
conceptualized by Spielberger (1966) as a transitory state or condition of the 
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organlsm that Is characterized by feelings of tension and apprehension and high 
levels of autonomic system activity. 

It could be hypothesized from the prior studies that some alteration In a 
computer-administered and -scored MMPI profile would result when compared to a 
traditionally-administered MHPI. Therefore, It seems of Import to determine the 
equivalence-validity of the MMPI as administered by computer vs. the traditional 
booklet administration of th^ test. 

Thus» It was the purpose of the present study to Investigate the equivalence- 
validity of the computerised MMPI as compared to a standard administration of the 
MMPI, I.e., the group booklet form. In addition, the affective and subjective 
reactions to both modes of administration were Investigated. 

Method 

Subjects 

A total of 63 female students^ participated in this experiment. All stu- 
dents were volunteers enrolled In psychology courses at Florida State University. 
Apparatus and Materials 

The IBM 1500 Computer-Assisted Instruction System (IBM, 1967) was used 
to present the MMPI. Students Interacted with the computer via terminals which 
consisted of (a) cathode ray tube; (b) light pen; and (c) typewriter keyboard. 
The terminals were located in an air-conditioned » sound-deadened room. 
MMPI 

As reporte/1 in more detail by Dunn et al. (in press), the automated version 
of the MMPI included all 566 items. Items were presented one at a time on the 
cathode ray tube. Students were instructed to depress "t^' for "true," "f" for 
"false," or "?" for "don't know" on the terminal keyboard to indicate their responses. 

^One student was dropped from the C/B group since her F scale score was 23. Scores 
this high generally indicate an invalid profile. 
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The statement !*Pre8S space bar to continue'* was Inserted arter each of the 566 
items to allow for accurate latency recordings. Latency served as a dependent 
measure in a study reported elsewhere (Duui et al^^ in press). The booklet form 
of the MMPI9 with accompanying answer sheets was also used. 

Twenty-slx scales were cosaputer scored. For the automated MHPI9 the IBM 
1500 system was used to score on-line the UMPI, whereas a Control Data 6 AGO was used 
to score off-line the booklet form of the VMPX. The scales scored included the 
13 original scales: Hypochondriasis (Hs)» Depression (D)» Hysteria (Ry), Psycho- 
pathic Deviate (Pd)^ Masculinity-Femininity (MF)t Paranoia (Pa)^ Psyches thenia 
(Pt)» Schizophrenia (Sc)^ Hypomania (Ma)^ Social Introversion-Extroversion (Si)^ 
Lie (L)^ Frequency (F)^ and Correction (K). 

Thirteen additional scales scored were: Social Maladjustment (SOC)^ De* 
pression (DEP)» Feminine Interests (FEM)^ Poor Morale (MOR), Religious Fundamentalism 
(REL)» Authority Conflict (AUT)^ Psychoticism (PSY)^ Organic Symptoms (ORG)^ 
Family Problems (FAM) » Manifest Hostility (HQS) » Phobias (PHO) » Hypomania (HYP) » 
and Poor Health (HEA) (Wiggins » 1969). 
A-State 

The A-State scale of the State-Trait Anxiety Inventory (STAI) developed 
by Spielberger» Gorsuch^ and Lushene (1970) was used to measure state anxiety. 
The short foi-in A-State scale consists of the five items from the total scale 
having the highest item-remainder correlation for the normative sample of the 
STAI. In a series of studies using this scale O^Nell (in press) and Leherissey^ 
0*Neil» and Hansen (1971) report alpha reliabilities ranging from .81 to .89. 
This scale was administered immediately before and Immediately after both booklet 
and computer administrations of the MMPI. When students were going through the 
computerized version » the anxiety scales were presented on-line. During the book- 
let administrations the anxiety scales were completed with paper and pencil. 



InstructioM for the pretest state anxiety scale asked students to des- 
cribe how they felt while taking the MMPI« 
Procedures 

Students who rlgned up to participate in this .experiment were randoaily as- 
signed to one of two groups: 

(a) Those who w> re to take the computer (C) version of the MMPI first and 
the booklet (B) version second (C/B); and 

(b) Those who were to take the booklet version of the MMPI first and the 
computer version second (B/C)* 

As students reported to the Computer-*Asslsted Instruction Center » they were 
asked to take a numbered card to Identify themselves* Students who had been ran- 
domly assigned to the C/B group were asked to have a seat in the terminal room. 
Those who had been randomly assigned to the B/C group went to a nearby classroom. 

The Procter for the C/B group then gave general instructions and inforaiation 
which included the number of items » the duration of the testt and some basic infor- 
mation abo':t the MMPI itself. In addition^ instructions were given for use of the 
cathode ray tube tetminal. 

The proctor with the B/C group gave the same ins true tionri< as those given 
to the C/B group » with the exception that the use of the cathode ray tube terminal 
was not discussed.. In addition^ the proctor with the B/C group distributed the 
test materials package. Includiid in this package » in this order » were: (a) STAI 
A-State pretest; (b) the MMPI booklet and an IBM answer sheet; and (c) the STAI 
A-*State posttest. 

All students were also Instructed that their responses would be confidential 
since they would be identified only by number. 

Procedures during the second session for both booklet and computer adminl-* 
strations were the same as in the first session> with the exception that studenta 



were not given the general information about the MMPI • After students finished the 
posttest A- State scale^ the experimenter held a debriefing session in which he ex- 
plained the experiment and answered questions about the test* 

Results 

In order to investigate whether or not the computerized MMPI is equivalent 
to a booklet administration of the same test and the students* affective reactions 
to mode of administration » the results are divided into five sections. The first 
section deals with the effects of mode and order of administration on MMEI profiles • 
The second section deals with the effects of mode of administration on MtPI scales • 
The third section deals with the equivalent validity of the two modes. The fourth 
section deals with the effects of mode and order of administration on state anxiety » 
while the fifth section deals with subject preference toward mode of administration. 
Effects of Mode and Order of Administration on MMPI Profiles 

Figure 1 pt<. Joints the mean T score profiles for the basic non-*K-corrected 
scales for (a ) the booklet administrations for each group (second for the C/B 
group » first for the B/C group); (b ) the computer administrations for each group 
(first for the C/B group and second for the B/C group ); (c ) the first administration 
(booklet for the B/C group and computer for the C/B group); and (d) the second 
administration (computer for the C/B group and booklet for the B/C group ). Figure 
2 presents the same profiles for the Wiggins scale. 

Discriminant function analyses were conducted in order to determine whether 
significant differences were present in these mean profiles . 

The results of the discriminant analyses on raw scores are shown in Table 1. 
It will be noted » as shown in Figure la^ that the relatively higher scores on the 
basic scales for the booklet administration for the B/C group resulted in signific^t 
profile differences for the booklet administration in eadi group. In addition » as 
may be seen in Figures Ic and d, respectively » the mean profiles for each group were 
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F Ratios Based Upc5n the Discriminant Function Tests on' the Mean Profiles 
of the B/C and C/B Groups for Both the Basic and Wiggins MMPI Scales 



Coznparison 





Booklet Administrations 


2.42* 


2.24*- 


(b) 


Computer Adtd.nistrations . 


1,14 


0.95 


(c) 


First Administration (C vs B) 


5.70** 


0c88 


(d) 


Second Administration (C vs B) 


5.67** 


1.13 



*p < .05; df » 26/35 
**p < cOl; df « 26/35 



significantly different for the first test administration as well as the second 
admlnist ration « There were, however, no significant differences between the 
computer administration for each group (Figure lb). In contrast, for the Wiggins 
scales there was only a significant difference between the mean profiles for the 
booklet administration to each group, (Figure 2a) ; the remaining comparisons were 
not significant. 

Effects of Mode of Administration on MMPI Scales 

While the foregoing discriminant analyses were primarily designed to ^ 
reveal any differences between the profiles of the mode of administration 
for each group cmd any order eff^c^ts, the following .analyses were concerned 
with pinpointing differences between the booklet and computer administrations 
of in^vidual scales within the same group o 

The mean profiles, in terms of T-scores, for the booklet and computer 
administration for each group are shown in Fl^gures 3a, b, c, and d. 
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Figure 3c. — ^Mean T-score profiles for the Wiggins scales for 
the C/B group . 
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Ccrrelated t-tests vere conducted on all of the MHPI scales and the results 
are presented In Table 2. Scales which showed significant changes between the 

TABLE 2 

Correlated t- Tests on Mean Raw- Scores for Each Test 
Administration for both Groups 







C/B Group 




B/C Grout) 






Knnir 1 At* 






Booklet 


^ Computer 




Dcaxe 


Ma An 

i^USCtti 


Me An 


t 


Mean 


Clean 


t 


lis 






5 14** 


7.17 


5.45 


3.00** 


D 


18.24 


18e61 


.67 


21.79 


19.38 


A. 09** 


Hy 


19.42 


18.94 


.82 


21.66 . 


16.69 


8.60** 


X>A 
ru 


m*^o 


1 ^ Aft 
u • fO 


2.38* 


15.97 


13.69 


3.70** 


Mf 


^7 • JO 


oy • 


.06 


39.97 


38o97 


1.74 


1>a 

Ira 


Q 


7 A7 


4 10** . 


10.83 


8.62 


3.17** 


Jr u 




16 ^6 


4.23** 


19.38 


16.97 


4.26** 


DC 


11 AA 

lit D<f 


t A 


4 92** 


18.59 


15.14 


3.02** 


na 


1 7 Aft 

1 # • HO 






18.97 


18.14 


1.38 


G-l 
oX 


OK 70 


9A m 


1 Rl 


33.66 


27.24 


9.08** 




^0 JZ 




2 77** 


2.69 


2.03 


2.74 


V 

r 




A 67 

•to W / 


3 44** 


6.45 


6.00 


.71 


V 


1A Sft 


9 ftft 


9 24** 


12o55 


9»00 


7.03** 


SIC 


11 27 


11.03 


.49 


13.66 


in 1 A 


5.24** 


Pd+O. &K 


20. 17 


17.48 


5.98** 


21.00 


17.31 


5.54** 


Pt+l.OK 


28.64 


26.24 


4.51** 


31.93 


25.97 


9.37** 


Sc+l.OK 


26.21 


23.91 


3.78** 


31.14 


24.14 


6.20** 


Ma+0.2K 


20.45 


19.15 


2.99** 


21.48 


19.86 


2.54*-. 


SOC 


9.48 


10.36 


1.69 


13.07 


13.21 


.34 


DEP 


6.73 


8.06 


2.78** 


9.90 


9.07 


1.37 


mi 


20.76 


20.70 


.21 


20.34 


20.69 


1.26 


MOR 


8.79 


9.79 


2.75* 


11.17 


10.24 


2.07* 


BEL 


5.55 


5.52 ■ 


.20 


5.34 


5.34 


.00 


AUI 


6.94 


7.18 


.68 


7.28 


7.83 


1.29 


PSY 


7.42 


9.12 


3.93** 


11.10 


10.69 


.65 


ORG 


3.33 


5.12 


4.22** 


7.14 


5.17 


3.36** 


FAM 


4.70 


4.91 


1.04 


6.03 


5.28 


2.15* 


HOS 


7.88 


8.12 


.59 


10.07 


10.10 


.05 


PHO 


7.48 


9.03 


4.62** 


9.34 


7.79 


3.38** 


HYP 


15.52 


15.45 


.21 


15.14 


16.03 


2.84** 


HEA 


3.30 


•4.58" 


3.86** 


6.28 


4.72 


4.16** 



*p < .05 
**p < .01 



15 

booklet and computer adnlhlstratlons In both groups were Hs^ Pd^ Pa» Pt> 
Sc> L» and K among the basic scales and MOR^ ORG^ PHO^ and HEA among the 
Wiggins scales. Scales which did not change significantly for either group 
were tiie basic Mf and Ma scales and thel71gglns SOC> ¥EM» KEL^ AUT» and HOS 
scales. Eight scales were significant for only on e groiq;>; thtree of thiise 
significant differences were in the C/B group and five in the.B/C group* 

Equivalent Validity 

Ihe comparisons in the prior sections tend to emphasize the differences 
between the booklet and computer profiles. Such differences > althou^.stttis** 
tically significant^ were not large in terms of absolute score change. 

Since the primary objective of this study was to determine the equiva-* 
lence*-validity of a computer administration of the MMPI coiq>ared to a booklet 
form» test*-ratest correlations were calci^lated between the coiq)uter and booklet 
scores for each scale. These correlations v^re computed separately for each 
group as well as the total sample. l!hese correlations are presented in Table 3. 

All of the test-re test coefficients were significant beyond the .01 
level and range from a low of » 47 to a high of .95 for the C/B group > a low 
of .42 to a hls(h of .f>3 for the B/C groups and a low of .45 to a high of .93 
for the total sample. 

Effects of Mode and Order of Administration on State Anxiety 

A five-item STAI A-State scale was administered both before and after 
each test administration. An overall analysis of variance was performed on^ 
tiiese data with groups » mode of administration » and pre-post as the inde- 
pendent variables and the A-State scores as the dependent variable. The triple 
Groups Mode x Pre-Post interaction was significant beyond the .001 level 
(F » 26.991 df • 1/168). The nature of this Interaction can be seen in Figure 4. 
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TABLE 3 

Test-Retest Coefficients for MHPI Scales for 
Each Group ' 



MMPI 



SCALE 


C/B Group 


B/C Group 


Both Groups 


Ha 




76 




D 


.73 


.85 


77 


Hy 


.57 


.73 


.52 


Pd 


.88 


.76 


.81 


Mf 


.80 


.73 


.7fi 


Fa 


.47 


.42 


.45 


Pt 


.87 


.92 


•85 


Sc 


.91 


.83 

• WW 


.80 

• WW 


Ma 


.85 


.73 

• » w 


• 80 

• WW 


Si * 


.80 


.90 

. ^w 


.83 

• WW 


L 


.83 

. WW 


*66 

e WW 


.75 


P 


.89 

. w^ 


.67 

• w# 


• 75 


K 


.80 

. WW 


.72 


• 77 


Hs+O.SK 


.51 


.68 

C WW 


•54 


Pd+0.4K 


.83 

. WW 


.68 

• WW 


.75 

• f w 


Pt+l.OK 


.79 


.86 


.78 


Scfl.OK - 


.82 


.78 


.75 


Maf0.2K 


.84 


.65 


.76 


SOC 


.84 


.93 


.89 


DEP 


.83 


.83 


.81 


mt 


.83 


.82 


.82 


MOR 


.92 


.87 


.88 


BEL 


.95 


.93 


.94 


AUI 


.85 


.85 


.85 


PSY 


.84 


.83 


.81 


ORG 


.79 


.75 


.67 


FAM 


.93 


.86 


.88 


H06 


.81 


.74 


.77 


PHO 


.88 


.86 


.81 


HYP 


.82 


.81 


.80 


HEA 


.73 . 


.35 


.73 
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FBEIEST 



1 I 

POSTIEST PRETEST FOSTIEST 



ikgure 4«--Mean STAI A-State scores for esch group tboth before 
and after the booklet and cooqputer adalfilstratloiis. 
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Subsequent analyses Indicated that the decrease in state axuciety scores during 
the coni^uter administration was statistically significant beyond the .01 level 
for both groups 9 while the change during the booklet administration was non- 
significant for each group* Further » it will be noted that the decrease in 
anxiety scores was much greater for the C/B group than for the B/C group* 

Correlated t-tests on the initial pretest scores for the opnputer and 
booklet administration revealed statistically significant initial differences 
(beyond the *01 level) for each group* There were» however > uo significant 
differences between the booklet and computer anxiety scor ^ at posttest A^State 
measuremtnt* 

Mode of Administration Preference 

In order to study the subjective preference for mode of administration » 
the subjects were asked to indicate which version of the MMPI they preferred 
after having completed both versions* For the C/B group (which answered ttie 
qtiestion after completing the booklet version) » 15 students preferred the 
computer version and 14 preferred the booklet version* For the B/C group i 
(which answered the question after completing the computer version) » 20 pre-*^ 
f erred the computer and 14 preferred the booklet* Overall » therefore » 35 
students preferred the coo^uter administration to 28 for the booklet* However » 
these differences in preference were not significant ^ *10; df - 1; *75)* 

Discussion 

Since regression towards the mean has often been observed upon MMPI 
retesting (Dahlstrom & Welsh» 1960) » the two groups were given the booklet 
and computer administrations in counter**balanced order* However » the results 
in the present study do not consistently reveal such regression toward the 
mean* While the computer profile is generally lower for the B/C grotq), the 
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booklet proflU is not consisttntly lover than the computer profile for the 
C/B group. 

^ch of the betveen-group variance seems to be attributable to. the 
relatively higher scores oa the booklet administration for the B/C group. 
It is also of some Interest to note that although the booklet scores for the 
B/C group were someiAat elevated over their computer scores on the basic scales 
the profilei? were quite similar and the test-retest coefficients vere conse-* 
quently rather high in spite of the absolute score differences. No consistent 
explanation for thase results, is imusdlately apparent. Likewise* the audety 
data Indicate that the B/C group* for 90m reason* vas a more state-anxious 
group Initially than vas the C/B group. However* the data do not demonstrate 
any consistent scale bias as a result of computer administrations of the MKPI. 

Although a number of scales showed significant differences between the 
computer and booklcit administration for the same group* the differences vere 
often not large (less than 2 score points) . Th« te3t«*retest coefficients were 
as hl^ as those reported for comparisons between the booklet and card form 
of the MMPI with a one-week Interval (e«g.* Cottle* 1950; Gilliland & Colgln* 
.1951; MacDonald* 1952). The median scale test-retest coefficient in those 
studies was .76* .67* and .721 The values also are in line with one«-week 
test-retest coefficients for the booklet alone; the median coefficient in a 
study by Gofer* Chance* and Judson (1949) was .68* and .81 in a study by 
Wlndle (1955). 

The finding that the coi^puter administrations to the two groups were 
not significantly different as were the booklet administretions at least 
implies that computer administration of personality tests results In more 
reliable measurement. Perhaps this is due to removal of such syetematic 
sources of variance as administrator personality* 
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The findings with respect to the anxiety data suggest that the first 
administration of the test Is xoet with more anxiety than Is the second • 
This finding supports a similar interpretation advanced by Dahlstrqm and 
Welsh (1960) to explain » at least in part» the regression towards the mean 
effect noted on re testing. In this study only the B/C group (higjhest A-State) 
shoved regression toward the mean. It is also apparent that, coopared to 
the booklet version, the computer initially produces rather hl^ anxiety 
levels. By the end of the test, however, no differences in. anxiety level, 
between the two modes was found. 

The fact that the coiq>uter profiles were not more /'deviant" than the 
booklet profiles suggests that perh^^s this initial anxiety level Is not 
maintained throughout .the test but quickly dissipates once the subject is 
comfortable responding to the computer; this explanation Is further supported 
by sxibjectlve reports which indicated that no difference in preferences 
existed towards the two modes of administration c The A*-State results are in 
contrast to Hedl et al. (1971), in which high levels of A-State were main- 
tained throughout a computerized Intelligence test. 

While the data reported here must be regarded as preliminary. It . 
appears that administration of a computerized MMPI is at least , as valid as 
administration via the booklet version. Further, computerized administration 
offers additional conveniences in terms of immediate on-line scoring and 
interpretation. 
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